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Abstract 

The tutorial videos contain an explanation of a learning material taught to students. The use of tutorial videos is 
common during the COVID-19 pandemic. This situation makes the teachers change the learning model into a 
video conferences or tutorial videos. However, the use of tutorial videos is often accompanied by opening other 
applications in parallel causing pop-up notifications to appear. The pop-up notification makes students not focus 
on the material explained in the tutorial videos. This raises the question of whether it will affect the learning 
process in understanding the learning material. Therefore, this study aimed to explore the influence of pop-up 
notifications on tutorial videos. Eye movements of all participants (N = 50) were recorded when viewing tutorial 
videos on various operating systems with or without the pop-up notification. Based on the results, after being 
shown a tutorial videos with a pop-up notification, participants paid attention to the pop-up notification. However, 
there were no significant differences in learning outcomes of students after viewing tutorial videos with or without 
pop-up notification. 


Keywords: Tutorial Video, Pop Up Notification, Eye Tracking 


1. Introduction 


The Industrial Revolution 4.0 has changed many aspects of human life, whether working, socializing, playing, or 
learning. Advances in information and communication technology make the use of technology enter various 
aspects of life, including in the field of education. The 21st-century learning process has the responsibility to 
develop 4C learning skills, namely Collaboration, Communication, Creativity, and Critical Thinking (Pertiwi & 
Rizal, 2020). The 21st-century learning challenge is to integrate knowledge-based learning processes with ICT 
skills (Information and Communication Technology) and scientific skills. Therefore, teachers are required to have 
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skills in creating effective and creative learning processes through technology-based interactive learning media 
(Hafizah, 2020). 


Technology-based interactive learning media combines several elements such as text, sound, graphics, images, 
and videos into a single unit to attract interest in understanding learning materials (Bustanil S et al., 2019). One 
of the technology-based interactive learning media is tutorial videos. Video-based learning has great benefits in 
influencing the future of the learning process to provide a place for students (Giannakos et al., 2014). The use of 
video supports student-centered learning so that it can be performed anywhere, in the classroom or at home, and 
at any time. 


The Covid-19 pandemic has made tutorial videos a solution in the learning process by requiring new, active, and 
constructive learning media. The use of tutorial videos has several weaknesses such as duration and pop-up 
notifications. Duration is very influential on understanding the learning material. According to (Guo et al., 2014), 
the ideal duration for tutorial videos is 6 — 9 minutes because, in that time span, the involvement of students 
towards tutorial videos was very high (around 75%). Short tutorial videos with good quality learning material 
require careful planning to design the concept in a concise and planned manner. Furthermore, the weakness of 
tutorial videos is the pop-up notification. This happens if students simultaneously open the application so that a 
pop-up notification appears when watching the tutorial videos making the attention breaks. 


However, studies related to the effectiveness of tutorial videos are mostly based on conventional methods by 
relying on answers from participants such as interviews or filling out questionnaires. In fact, even if the questions 
have been structured in such a way, the potential for bias could occur due to the potential for faking good (Fairuz 
Izdihar et al., 2019)or a condition in which participants gave inaccurate answers during interviews or filled out 
questionnaires due to various reasons, such as efforts to maintain self-image. Therefore, to minimize the potential 
for bias and increase objectivity, it is necessary to consider the use of technology such as human eye trackers. 
Through infrared rays emitted into the eye and reflected back to the instrument (Okuyama et al., 1990), the human 
eye tracker can be an alternative to assess eye response when viewing tutorial videos. 


Real-time visual attention analysis such as eye movement, viewing duration, frequency of eyes looking back at 
the same area, and so on can also be performed (Reingold, 2014). The use of an eye tracker as an evaluation tool 
is able to accommodate human behavior to improve the interface appearance of various products and to increase 
understanding of human behavior (Diego-Mas et al., 2019). The eye tracker utilizes infrared sensors and cameras 
to track eye movement when viewing a display (Hessels & Hooge, 2019). This technology can detect focus and 
attention by tracking eye movement (Carter & Luke, 2020). Visual attention is automatic, besides being controlled 
by the human brain without realizing it, it can also be influenced by mental status (Van Eymeren, 2016). Therefore, 
it is very important to know the visual attention of students to a tutorial video and to detect interest and boredom 
towards tutorial videos. 


2. Method 


The participants of this study were Informatics Engineering students (N=50). Participants were divided into two 
groups, namely pop up notification (N=25) and non-pop-up notification (N=25) with several inclusion criteria 
namely not having a visual impairment, not wearing glasses, not wearing contact lenses, not color blind, not having 
strabismus, not having slanted eyes, and not wearing eyelashes. 


In this study, participants were given a pre-test regarding the operating system based on the working process. 
Then, the data were collected using the Tobii Pro X2-30 HET (Human Eye Tracker). Participants were sitting on 
chairs in a relaxed condition with a distance of 60 — 70 cm from the monitor screen to their eyes. During recording, 
participants were asked to minimize head movement (Figure 1), while the room lighting was set not too bright or 
dim (Edison et al., 2021). Before recording, the HET instrument calibration was carried out to ensure the precision 
of the eye position against the object seen on the monitor screen. This study used a go/no-go experiment design 
(Fooken & Spering, 2019). During data collection, participants were shown a black screen for five seconds 
followed by a 10-minute learning video and closed again with a black screen for five seconds (Figure 2).. 
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Figure 1: Participants sit relaxed facing the monitor screen with a distance of 60-70 cm 


Table 1: Screen Condition 


Screen Condition with Pop-Up Screen Condition With Pop-Up 
Notification Notification 
Blank Space 2 Blank Space 2 
Tutorial Video 1 Tutorial Video 


MESS Video Pembelajran 


Blank Space 
rare MEE Blank Space 


Videc 


Pembelajaran 


Blank Space 


Figure 3: Go/no-go simulation 


Table 2: Tutorial Video with and without Pop-Up Notification 


Tutorial Video 
Pop-Up Notification 


(es Sistem Operasi Berbasis Cloud 


Sistem operasi berbasis cloud merupakan jenis 
sistem operasi yang dirancang untuk beroperasi dalam 
komputasi berbasis awan dan virtualisasi ruang lingkup. 
Tipe layanan cloud: 

+ Infrastructure-as-a-Service (laaS) 
+ Platform-as-a-Service (PaaS) 
+ Software-as-a-Service (SaaS) 
+ Functions-as-a-Service (FaaS) 


No Pop-Up Notification 
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oO Sistem Operasi Berbasis Cloud 
= g 


A yi i i I 


Sistem operasi berbasis cloud merupakan jenis 
sistem operasi yang dirancang untuk beroperasi dalam 
komputasi berbasis awan dan virtualisasi ruang lingkup. 
Tipe layanan cloud: 

* Infrastructure-as-a-Service (laaS) 
+ Platform-as-a-Service (PaaS) 

+ Software-as-a-Service (SaaS) 

+ Functions-as-a-Service (FaaS) 


The stimulus used was in the form of tutorial videos on operating system courses with pop-up notifications and 
non-pop-up notifications for 10 minutes. In the stimulus, the Region of Interest was used to mark the object. The 
tutorial video with pop-up notification was divided into 2 ROIs namely the Learning area and Pop up area. While 
in the tutorial video without pop-up notification, ROI was only found in the learning area (Table 3). Furthermore, 
a post-test regarding the operating system and its working procedure was given. 


Based on the pretest and post-test, each participant was given 10 points if the answer was correct on each question, 
and 0 points if the answer was wrong on each question. Data underwent T-test analysis using IBM SPSS 28. HET 


results were analyzed using Tobii Pro Studio to obtain a gaze plot and time to first fixation (3.4.8). 


Table 3: ROI of Tutorial Video with and without Pop-Up Notification 


ROI of Tutorial Video 
Pop-Up Notification 


oO Sistem Operasi Berbasis Cloud 


> Learning Area 


= | > Pop Up Notification 


No Pop-Up Notification 


Co Sistem Operasi Berbasis Cloud 


> Learning Area 
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3. Results 


There were 50 participants following the overall recording procedure. The procedure starts with explanation, 
calibration, and simulation. The time required by each participant to complete the entire process was approximately 
45 minutes. 50 participants were divided into two groups; Pop-Up Notifications and Non-Pop-Up Notifications. 


Table 4: Shapiro-Wilk Normality Test 


Pop-Up Notification Group Non Pop-Up Notification Group 
Pretest Pretest 


0 5 10 5 0 5 i) 5 10 5 20 25 
Statistics= 0.929, p=0.081 Statistics= 0.912, p=0.034 
Sample follows Gaussian (fail reject HO) Sample does not follow Gaussian (reject HO) 
Pop-Up Notification Group Non Pop-Up Notification Group 
Post-test Post-test 
° 
° 
6 
ee 
e > 
eee 
ee % % e 
° ee 
7 10 5 20 25 0 5 10 15 20 25 
Statistics= 0.955, p=0.321 Statistics= 0.928, p=0.078 
Sample follows Gaussian (fail reject HO) Sample follows Gaussian (fail reject HO) 


Table 3 shows the results of the Shapiro-Wilk Normality Test on pre and post-test of pop-up notification and non- 
pop-up notification groups. If the significance value (p) in the Shapiro-Wilk test > =0.05, the data were normally 
distributed. Based on these tests, the results obtained from each group, pretest of pop-up notification group had p= 
0.081 showing no significant influence of pop-up notification on learning outcome with normal distribution. 
Pretest of non-pop-up notification group had p= 0.034 showing the significant influence of pop-up notification on 
learning outcome and the data were not normally distributed. Post-test of pop-up notification group had p=0.321 
and non-pop-up notification had p= 0.078 showing no significant influence of pop-up notification on learning 
outcome with normally distributed data. 
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Table 5: T-test analysis 


No Data 1 Data 2 Result Abs(t_stat) <= cv p> alpha 
t =-0.200 
Pretest of Pop-Up | Pretest of Non Pop-Up | p =0.843 
1 ; . : ; True True 
Notification Group Notification Group df = 24 
ev=1.711 
t= 0.529 
) Post-test of Pop-Up Post-test of Non Pop- | p= 0.602 True True 
Notification Group | Up Notification Group | df= 24 
ev=1.711 


T-test analysis showed that the pretest result of pop-up notification and non pop-up notification groups was 0.843, 
while the post-test was 0.602 meaning no significant influence on learning outcome (Table 4). 


Table 6: Gaze Plot and Time To First Fixation 


Pop-Up Notification 


No Pop-Up Notification 


Gaze Plot 


Gaze Plot 


oO Sistem Operasi Stand Alone 


i 


Sistem Operasi Stand Alone, sistem operasi yang dapat 
berdiri sendiri, dan bisa digunakan baik single user maupun 
multi user, Tugas dari sistem operasi ini adalah: 

+ Memberikan dukunqa' ing baik untu irk 
= Memberik: 43 + 36 

+ Memberik: i e. 
» Memberikan 


ti 
itu 
ang baik untuk akses i ate 


[e ere wie Alone 


a 


La work processing. 
* Memberikan dukungan yang baik untuk spreadsheet. 
+ Memberikan dukungan yang baik untuk game. 

+ Memberikan dukungan yang baik untuk akses internet, 


ang dapat 
yy maupun 


Pop Up Notification 


No Pop Up Notification 


Time To First Fixation 


Time To First Fixation 


Second 


Learning Area 


Pop Up Notification 


Second 


233,71 


Learning Area 


Environment 


Furthermore, based on HET analysis, the time to first fixation matrix showed the time it took respondents (or the 
average of all respondents) to see the regio of inters. Of the 9 pop-up notifications, 8 notifications were seen 
directly by participants when they appeared. The gaze plot on the pop-up notification can see the order of the eye 
movement, when the pop-up notification appears, the gazes go directly to the pop-up, then switch to the learning 
area. This is evidenced by the time to first fixation matrix showing that participants see the pop-up notification 
before the learning area. Then in the non-pop-up notification group, gaze plot and time to first fixation matrix 
showed that participants were more focused on learning areas than outside the learning area (Table 5). This proves 
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that when the pop-up notification appears, participants go directly to the pop-up notification, although only for a 
moment but when compared with the results obtained from the post-test, participants who saw tutorial videos with 
pop-up notifications had an influence, but when compared to the pretest with the non-pop-up notification group, 
both did not show significant results. This shows that pop-up notifications had no significant influence on learning 
outcomes. 


4. Discussion 


This study was carried out on Informatics Engineering students who have never taken an operating system course. 
All participants followed the overall study procedure from pretest, tutorial videos, and post-test. 


The students were shown tutorial videos with pop-up notifications. Based on the t-test on pre and post-test, the 
pop-up notification had no significant influence on learning outcome. Then, the first thing participants saw was a 
pop-up notification. However, the duration and frequency of seeing pop-ups are low. As with Gaze-Plot, from 9 
pop-up notifications displayed, 8 pop-up notifications were seen immediately when they appeared and only | pop- 
up notification was not seen immediately. Furthermore, in the gaze plot analysis, one - three times the initial eye 
movement of the participants looked at the pop-up notification. While the rest was focused on the area in the pop- 
up notification. Specifically, the data showed that the time duration when the notification was shown 
simultaneously with the tutorial videos, eye-stopping when viewing the pop-up area was faster than viewing the 
learning area. This phenomenon is in line when comparing the visual attention of students when shown tutorial 
videos showing instructors without showing instructors where participants tend to stare more at the face than the 
material. This makes the participants not focus on the material presented (van Wermeskerken et al., 2018). 


The eye tracker is used as an instrument to analyze online learning during the covid-19 pandemic. On average, 
students are given material through video tutorials. The eye tracker can find out what things can be annoying if 
given a video about learning materials. Unlike the case with questionnaires or interviews where someone has the 
opportunity to think about the answers given, in other words, someone can determine actions consciously, while 
visual attention is automatic or done unconsciously. This phenomenon occurs when viewing an object, making 
visual attention analysis using human eye tracker applicable in various fields such as advertising, where the visual 
attention of consumers when viewing an advertisement becomes the input for the development of new types of 
advertisements. (Punde et al., 2017).In addition, the human eye tracker can also be used to find out what smokers 
see when images of cigarette packs are displayed, where smokers tend to focus on cigarette brands, while non- 
smokers focus more on health graphic warning images. (Edison et al., 2021). 


Visual attention such as eye movement, viewing duration of an object, the number of blinks of the eye is automatic 
without being aware of it by the brain or psychologically. With this human eye tracker technology, psychological 
phenomena in a person when shown an image also have the potential to. 


When shown tutorial videos through the monitor, participants focus more on the pop-up notification than on the 
learning area but only for a moment. Therefore, this study shows that running the application simultaneously while 
watching the tutorial videos then displays a pop-up notification generated by the application showing no significant 
influence. In the future, it is possible to compare the location of the pop-up position on the bottom right with 
appearing in the middle of the monitor layer. Short tutorial videos with good quality learning material require 
careful planning to design the concept in a concise and planned manner. 


5. Conclusion 


After being shown a tutorial videos with a pop-up notification, participants paid attention to the pop-up 
notification, but only for a moment and then back to the learning area. This is confirmed by pretest and post-test 
results using t-test analysis. There were no significant differences in learning outcomes of students after viewing 
tutorial videos with or without pop-up notification participants. It is still recommended to turn off notifications 
when doing online learning so that students can focus on learning in accordance with the basic competencies 
achieved. 
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